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Abstract 


In 2006, the easyCBM reading assessment system was developed to support the progress 
monitoring of phoneme segmenting, letter names and sounds recognition, word reading, 
passage reading fluency, and comprehension skill development in elementary schools. 
More recently, the Common Core Standards in English Language Arts have been 
introduced as a framework for outlining grade-level achievement expectations, to guide 
reading instruction and assessment across the United States. The purpose of this study 
was to examine the extent to which the easyCBM Phoneme Segmenting assessment 
contains items aligned with the Common Core Standards. Alignment analyses focused on 
expectations stated in Standard Two, Foundational Skills, for kindergarten and first 
grade. One hundred and sixty-nine words from the Phoneme Segmenting assessment 
were reviewed, comprising 98.2% of the unique words on the kindergarten assessment, 
and 100% of the unique words on the first grade assessment. Study results indicated that 
kindergarten Phoneme Segmenting items were weakly or not aligned with any of the 
Standard Two expectations. First grade Phoneme Segmenting items were strongly 
aligned with one Standard Two expectation (i.e., segmenting single-syllable words), but 
weakly or not aligned with the others. Reviewer errors and specific test item issues found 


were also discussed. 
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In 2010, the Common Core State Standards (CCSS) were introduced as a way to 
incorporate national standards for instruction and assessment. Because the easyCBM reading 
assessment system (Alonzo, Tindal, Ulmer, & Glasgow, 2006) was developed four years prior to 
the advent of the Standards, we examined the extent to which existing Phoneme Segmenting test 
items are aligned with the skills and knowledge expected of students, as reflected in the CCSS 
for kindergarten and first grade students (National Governors Association Center for Best 
Practices & Council of Chief State School Officers, 2010). 

The easyCBM Phoneme Segmenting assessment was designed to exclusively measure 
phoneme segmenting across 20 forms within a grade. A preliminary examination of the CCSS 
indicated that only some components of Standard 2 would be relevant for analysis because they 
specifically refer to phoneme segmentation activities (see RF.K.2 D, and RF.1.2 C and D below). 


Reading: Foundational Skills Standard 2 

RF.K.2. Demonstrate understanding of spoken words, syllables, and sounds (phonemes). 

A. Recognize and produce rhyming words. 

b. Count, pronounce, blend, and segment syllables in spoken words. 

c. Blend and segment onsets and rimes of single-syllable spoken words. 

D. Isolate and pronounce the initial, medial vowel, and final sounds (phonemes) in 
three-phoneme (consonant-vowel-consonant, or CVC) words , 1 (This does not include 
CVCs ending with /!/, /r/, or /x/.) 

E. Add or substitute individual sounds (phonemes) in simple, one-syllable words to make 
new words. 

RF.1.2. Demonstrate understanding of spoken words, syllables, and sounds (phonemes). 

A. Distinguish long from short vowel sounds in spoken single-syllable words. 

b. Orally produce single-syllable words by blending sounds (phonemes), including 
consonant blends. 

c. Isolate and pronounce initial, medial vowel, and final sounds (phonemes) in spoken 
single-syllable words. 

d. Segment spoken single-syllable words into their complete sequence of individual 
sounds (phonemes). 
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Phoneme Segmenting Alignment 

Traditionally, examinations of test and standards alignment entail the use of a rating scale 
to judge the degree to which items correspond with standards of interest. Because of our narrow 
focus on only particular components of Standard 2 within each grade, we used an alternative 
method for evaluating alignment that we believed would be a more infonnative approach than 
traditional methods. Specifically, we had reviewers provide item-level information associated 
with each component of interest, so that we could simultaneously examine reviewer agreement 
(similar to typical alignment methods) and background knowledge underlying reviewer 
responses. 

For example, according to the first grade CSS Standard 2, students are expected to 
engage in segmenting activities using single-syllable words (see components A & C). Rather 
than have reviewers judge how well Phoneme Segmenting test items reflected single-syllable 
words (as in typical alignment), we had reviewers identify the number of syllables for each word 
test item, which we used to calculate the proportion of single-syllable words. We reasoned that 
the more test items resembled expectations outlined in the CCSS, the stronger the alignment 
between the assessment items and the standards. 

We also examined the background knowledge of reviewers by having them decompose 
the same words into their constituent syllable parts. We anticipated that this step would inform 
our understanding of reviewer agreement concerning the number of syllables identified for each 
item by revealing the strength of word syllabication knowledge among participants. Because 
previous research has indicated that beginning literacy knowledge may be weak among K-3 
teachers (Cunningham, Perry, & Stanovich, 2004), we wanted to account for weak knowledge 


that might lead to underestimates of correct agreement among reviewers. For example, although 
a teacher reviewer may accurately provide the number of syllables in a word, s/he may not 
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precisely know the sound parts that comprise each syllable, as a consequence of incomplete 
knowledge. By inspecting reviewers’ responses in this way, we were able to better capture the 
knowledge base underlying the infonnation provided. We followed the same logic and practice 
for obtaining phoneme information about test items. 

An additional benefit to using this alternative alignment method was that it illuminated 
how well test items reflected instructional practice. Items found to be problematic for teacher 
reviewers likely indicate sources of test error variance that warrant further investigation because 
such items potentially measure something other than student phoneme segmenting skill. For 
example, these items may reflect factors that extend beyond the measurement of student skill 
such as limited exposure to particular types of words for segmenting practice, or gaps in teacher 
knowledge that diminish measurement precision and offer less useful test information for making 
instructional decisions. Thus, items identified as problematic may suggest a test element that 
could potentially weaken the link between CCSS expectations, reading measurement outcomes, 
and instructional implications for helping students demonstrate expected knowledge and skills. 

Methods 

Participants 

Between November 201 1-March 2012, 9 teachers were trained to do the following as part 
of their item review: 

a) Indicate the number of syllable word parts and decompose each word into syllables. 

b) Indicate the number of phonemes and segment each word into phonemes. 

Teacher participants comprised four current kindergarten teachers with M= 3.25 years of 


experience (range =1-5 years) and three current first grade teachers withM= 1 1.33 years of 
experience (range = 3-25 years). In addition, two current special education teachers also 
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participated in this study, with M = 3.5 years of experience. The majority of the participants were 
teaching in the state of Oregon; two participants were teaching in Iowa and Ohio. 

Procedures 

Because a majority of items on the easyCBM Phoneme Segmenting assessment were 
repeated across kindergarten and first grade level test forms, to minimize redundancy of item 
reviews, words were combined into a single item review pool and reviewed once by each of 
three reviewers. The unique word count, instead of the total word count (which would include 
repeated word occurrences across all test forms), was used for reported calculations. Three words 
were unexpectedly not reviewed (“scale” and “trip” on kindergarten form 10, and “straight” on 
kindergarten form 8). Therefore, 170 words were reviewed, split across seven review measures 
of approximately 25 words each (see Table 1). This comprised 98.2% of the unique words from 
the kindergarten easyCBM assessment (i.e., 173 words), and 100% of the unique words used 
from the first grade assessment (i.e., 159 words). 

Each teacher reviewed items on two measures for a total of approximately 50 words (see 
Table 2; each teacher is represented by a different alphabetic letter, followed by his/her current 
instructional placement). Where possible, teacher reviewers with different instructional 
placements were counterbalanced across measures to control for potential differences in 
background due to prior teacher training and classroom instruction experience. 

Reviews were conducted using an online review website designed for evaluating test item 
bias, sensitivity, and alignment to standards. As part of the review, teachers were shown one 
word at a time and provided requested information for each test item (e.g., number of syllables, 
syllable decomposition, number of phonemes, phoneme decomposition). A sample review 
question for the word cover is shown below. 
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TEST ITEM 
cover 

ITEM REVIEW QUESTIONS 
Number of syllables: 

Ol 62 03 04 

Type each syllable part into the text box. Separate syllables using a slash (/). 

Example: dinner: din / ner 


Additional text box features enabled teachers to list any resources that were used to support their 
reviews, as well as the opportunity to express concerns (either regarding the certainty of their 
responses or particular items). Although few teachers listed resources, the following were 
reported as resources used with challenging items: Merriam-Webster dictionary, Wikipedia, 
Direct Instruction teacher manual, an English phoneme chart (retrieved online), and consultation 
with a reading specialist at the reviewer’s school. 

Pre-review training was required of participants through a 30-minute webinar, in which 
teachers were briefed about study logistics, relevant reading tenninology, and use of the online 
review system. Teachers were given one month to review items, commencing the day after 
training. They were compensated for their completed review with a $25 gift card. 

Reviewer response accuracy was examined by the first author, a reading researcher and 
former special education teacher. The following resources were referenced as support materials 
during the data analysis phase of the project: an online dictionary, easyCBM Phoneme 
Segmenting answers (not available to reviewers), and a language text (Moats, 2000). 
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Results 

Although 170 words were reviewed, one word was dropped from analysis because two 
reviewers misread the word (i.e., “yam” was misread as “yarn”), reducing the number of 
kindergarten words reviewed to 169 and first grade words reviewed to 158. Only 1 1 words were 
unshared between grades, reflecting significant overlap in words used on measures across both 
grades. 

The majority of words reviewed were one-syllable words (i.e., 78.7%). Table 3 displays 
the frequency distribution of one-syllable words across grade -level forms, and also shows the 
large item overlap across both grades. The remaining 36 words reviewed comprised two-syllable 
words (e.g., apron, city, flowing, open, and repeat). The number of word phonemes ranged from 
two to seven. Of the 169 words reviewed, 3 words contained two phonemes, 66 words contained 
three phonemes, 74 words contained four phonemes, 17 words contained five phonemes, 8 words 
contained six phonemes, and 1 word contained seven phonemes. 

Alignment-specific findings by grade are presented below, with text bolding used to 
highlight findings and conclusions specific to each Standard expectation component under study. 
We also present findings related to reviewer agreement and problematic words, and conclude 
with a brief summary. 

Alignment with Kindergarten Standard 2 Components 

Standard RF.K.2D: Isolate and pronounce the initial, medial vowel, and final 

sounds (phonemes) in three-phoneme (consonant-vowel-consonant, or CVC) words. 1 

(This does not include CVCs ending with /l/, Irl, or /x/.). 

Of the 169 kindergarten words reviewed, 3 words contained two phonemes (2.2%), 65 


words contained three phonemes (48.5%), and 66 words contained four or five phonemes 
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(49.3%). Only seven of the three-phoneme words were identified as CVC (i.e., cup, fit, hid, 

lag, mom, tap, win), and one word (box) was identified as a three-phoneme word that ends 

with a IV, /r/, or hJ. The three-phoneme CVC words comprised less than 5% of the unique 

words reviewed. The extent to which students “isolate and pronounce” phoneme sounds cannot 

be directly assessed by the Phoneme Segmenting assessment; however, indirectly, the ability to 

isolate the initial, medial vowel, or final sounds can be inferred from student errors when the test 

is administered (e.g., an error analysis could indicate whether a student demonstrated sensitivity 

for isolating beginning or final sounds). Therefore, weak evidence of alignment with this 

kindergarten Standard expectation was found. In general, the level of segmenting difficulty 

exceeded the expectations outlined in Standard 2 (i.e., nearly half of the kindergarten items 

contained more than three phonemes). 

Alignment with First Grade Standard 2 Components 

Standard RF.1.2.C: Isolate and pronounce initial, medial vowel, and final sounds 
(phonemes) in spoken single-syllable words. 

Of the 158 first-grade words reviewed, 127 were single syllable words (79.9%). 

However, as previously mentioned, the extent to which students demonstrate the ability to isolate 
only initial, medial vowel, or final sounds can only be inferred indirectly from student errors 
(e.g., when attempting to segment “shout” only responding with the /sh/ sound would provide 
informal evidence that while the student can isolate the first phoneme, s/he lacks sufficient 
phonemic awareness to fully segment the word). Therefore, weak evidence of alignment with 
this first-grade Standard expectation was also found. 

Standard RF.1.2D: Segment spoken single-syllable words into their complete 
sequence of individual sounds (phonemes). 
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As previously mentioned, nearly 80% of the items on the first grade assessment 
were single syllable words and all test items required students to segment these words into 
their complete sequence of individual sounds. Of these words, 3 contained two phonemes 
(2.2%), 62 contained three phonemes (46.3%), and 69 contained four or five phonemes (51.5%). 
Therefore, strong evidence of alignment with Standard 2D in first grade was found. 

Reviewer Agreement 

In general, there was moderate agreement among reviewer responses. However, one 
reviewer committed mistakes on more than half of the words she reviewed, which weakened the 
overall percent of correct agreement among reviewers. Therefore, the percent of agreement 
found among all three reviewers, as well without this reviewer, are presented. 

The number of syllables comprising each test item was correctly identified 75.6% of the 
time (97.0% with reviewer removed). Correct syllable decomposing occurred 68.0% of the time 
(94. 1% with reviewer removed). The number of phonemes comprising each test item was 
correctly identified 75.1% of the time (86.4% with reviewer removed). Correct phoneme 
segmenting occurred 75.7% of the time (89.9% with reviewer removed). Table 4 displays the 
types of syllabication and phoneme segmenting errors committed by reviewers. 

The extent to which errors were similar across reviewers varied (see Table 4). In some 
cases there was agreement regarding the quantity of sound parts, but how the words were 
composed of either syllables or phonemes indicated different reviewer misconceptions. 
Combined, these findings suggest that these particular test items are especially challenging to 
segment, and may be either too difficult for students or not relevant to beginning reading 
instruction intended to teach word segmenting at the syllable or phoneme level. 


Problematic Words 
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Thirty-one words (18.3%) were identified as problematic for the following reasons: a) 
non-representativeness of typical kindergarten or first-grade vocabularies (i.e., developmentally 
or instructionally inappropriate words); b) extreme segmenting difficulty as evidenced by teacher 
errors or the quantity of sound parts; and/or c) pronunciation concerns that could adversely 
impact student response accuracy. Five words were identified as potentially problematic by 
teachers: bide, gnat, omen, snare, and thoughtless. The first author identified additional words 
based upon reviewer accuracy and qualitative assessment of items using the previously 
mentioned a-c criteria: box, chalk, city, flowing, futile, glitch, hire, hour, jobless, lame, leaping, 
lice, omit, rear, remote, rental, repeal, Roman, scan, sealer, slab, spouse, theft, trait, wrath, and 
wren. In selecting these words, careful attention was paid to the appropriateness of segmenting 
these items and the validity of inferences made regarding how well students can segment 
developmentally and instructionally appropriate words. Extremely difficult words to segment 
that are not practiced as part of typical instruction to build phoneme awareness, or words 
unfamiliar to student examinees, add construct irrelevant variance to the measurement (as 
discussed in the introduction), and are concerns to be considered when examining the 
appropriateness of measure use in relation to the CCSS and instructional practices. 

Conclusion and Study limitations 

The easyCBM Phoneme Segmenting assessment was developed prior to the adoption of 
the CCSS. Consequently, the assessment was not designed to directly align to expectations set 
forth by the CCSS. The results from this study suggest that the first grade Phoneme Segmenting 
assessment is most aligned with the Standard 2, Component D. Nearly 80% of the unique words 
used for the first grade assessment were identified as single-syllable words, and the test (by 
design) requires students to demonstrate that they can segment words into their complete 
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sequence (consistent with the Standard 2D expectations). However, alignment with the other 
components was either weak or absent. In contrast, only weak or absent evidence of alignment 
was found for the kindergarten Phoneme Segmenting assessment. 

A number of study limitations must be considered in conjunction with reported 
conclusions. First, the number of item reviewers per word was three, which limits the 
generalizability of these findings to the larger population of kindergarten, first-grade, and special 
education teachers. Therefore, we consider results related to reviewer agreement and problematic 
items preliminary. In addition, although we recruited participants nationally, the final sample for 
this study over-represented the state of Oregon. Teacher drop-out and our inability to recruit 
replacements with the same teaching background from other states in a timely fashion weakened 
the alignment review design. 
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Table 1 . K/l Phoneme Segmenting Test Items by Measure Reviewed 


Ml 

M2 

M3 

M4 

M5 

M6 

M7 

able 

cows 

globe 

letter 

owner 

shade 

sneaky 

apron 

crew 

glows 

lice 

pack 

shed 

tail 

bake 

crowd 

glum 

lime 

paid 

shiny 

tap 

bend 

crumb 

gnat 

listen 

pounce 

shirt 

theft 

bent 

cup 

graph 

loaf 

race 

shout 

theme 

bide 

desk 

green 

lobe 

rack 

silent 

then 

black 

dimmer 

hid 

lump 

raid 

skin 

these 

blood 

dive 

hire 

main 

ramp 

slab 

tint 

blur 

down 

home 

male 

rank 

sled 

trait 

boast 

draw 

hound 

mass 

rant 

slowly 

trap 

bold 

drip 

hour 

mean 

read (ee) 

smile 

treated 

bone 

dunk 

huddle 

mess 

rear 

snag 

wear 

bow 

fair 

include 

metal 

regrow 

snail 

win 

box 

fit 

inspire 

mine 

release 

snake 

wind 

brace 

float 

jobless 

mint 

remote 

snare 

word 

brand 

flowing 

jokes 

moat 

rental 

sneak 

wraps 

bunk 

fold 

jump 

mom 

repeal 

snout 

wrath 

cane 

foul 

kettle 

must 

repeat 

soak 

wren 

chalk 

found 

knives 

neater 

Roman 

span 

wrist 

chap 

fray 

lag 

nine 

rule 

spoken 

yam 

chef 

free 

lamb 

nurse 

scan 

spouse 

n 

city 

futile 

lame 

omen 

scoop 

stack 

clearest 

gift 

leaping 

omit 

seal 

strap 

clink 

glass 

left 

open 

sealer 

street 

clown 

glitch 

lend 

ouch 

send 

thoughtless 
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Table 2. Teacher Reviews by Measure 


Ml 

M2 

M3 

M4 

M5 

M6 

M7 

A- 1st 

B-lst 

A- 1st 

B-lst 

D-lst 

E-K 

D-lst 

C-K 

C-K 

E-K 

F-K 

F-K 

G-lst 

G-lst 

H-SPED 

I- 1st 

J-SPED 

H-SPED 

I- 1st 

J-SPED 

K-K 
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Table 3. Distribution of One Syllable Words Comprising the Kindergarten and First Grade 
Assessments 


Frequency of Presentation 
Within Grade-level Forms 

Kindergarten 

First Grade 

1 Form 

Bake 



Bend 

Bend 


Black 

Black 


Blur N z 54 

Blur N - 53 

Chalk 

Chef 


Clown 

Clown 


Cows 

Cows 


Crumb 

Crumb 


Down 

Dunk 

Down 


Float 

Float 


Fold 

Fold 


Found 

Found 

Free 


Gift 

Gift 


Glass 

Glass 


Graph 

Green 

Graph 


Hound 

Hound 


Hour 

Hour 


Jokes 

Jokes 


Jump 

Knives 

Jump 


Lag 

Lag 


Lend 

Lend 


Lice 

Lice 


Loaf 

Loaf 


Lobe 

Lobe 


Lump 

Lump 


Nine 

Nine 

Race 


Rank 

Rank 


Rant 

Rant 


Rear 

Rule 

Rear 


Scan 

Scan 


Send 

Send 
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Skin 

Skin 


Sled 

Sled 


Snag 

Snag 


Snake 

Snake 


Snare 

Snare 


Snout 

Snout 


Span 

Span 


Street 

Street 


Theft 

Theft 


Then 

Then 


These 

These 


Tint 

Tint 


Trait 

Trait 


Trap 

Trap 


Wear 

Wear 


Win 

Win 


Word 

Word 


Wraps 

Wraps 


Wren 

Wren 


Yam 

Yam 

2 Forms 

Blood 

Blood 


Boast N - 41 

Boast N - 34 


Bold 

Bold N ' 34 


Bone 

Bone 


Brace 

Brace 


Brand 

Brand 


Bunk 

Bunk 


Chalk 



Chap 

Chap 


Chef 



Clink 

Clink 


Crowd 

Crowd 


Desk 

Desk 


Dive 

Dive 


Drip 

Drip 


Foul 



Fray 

Fray 


Free 



Globe 

Globe 


Glum 



Hire 

Hire 


Lame 

Lame 


Left 

Left 


Main 

Main 


Mass 

Mass 


Mess 
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Table 4. Reviewer Errors by Syllable and Phoneme Word Part 


Word Part 

Correct 

Decomposing/ 

Segmenting 

Response 

Reviewer 1 errors 

Reviewer 2 errors 

Reviewer 3 errors 

Syllable 


bake 

1- bake 


3-b/a/ke/ 

2-b/ake/ 

hour 

1-hour 

2-hou/r/ 

2-ho/ur/ 


male 

1-male 

2-ma/le/ 


2-ma/le/ 

Phoneme 


box 

4-b/o/k/s/ 

3-b/o/x/ 

2-b/o/x/ 

2-b/ox/ 

chalk 

3-ch/al/k/ 

3-ch/a/lk/ 

4-ch/a/l/k/ 


crowd 

4-c/r/ow/d/ 


3-c/row/d/ 

3-cr/ow/d/ 

snare 

4-s/n/a/re/ 

3-s/n/are/ 


3-s/n/are/ 
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